1. Convolutional Neural Networks
2. Transformers
3. BERT

• Intuition: Neural network with specialized connectivity structure
  • Stacking multiple layers of feature extractors, low-level layers extract local features, and high-level layers extract learn global patterns.
• There are a few distinct types of layers:
  • Convolution Layer: detecting local features through filters (discrete convolution)
  • Pooling Layer: merging similar features

• The core layer of CNNs
• Convolutional layer consists of a set of filters
• Each filter covers a spatially small portion of the input data
• Each filter is convolved across the dimensions of the input data, producing a multidimensional feature map.
• As we convolve the filter, we are computing the dot product between the parameters of the filter and the input.
• Deep Learning algorithm: During training, the network corrects errors and filters are learned, e.g., in Keras, by adjusting weights based on Stochastic Gradient Descent, SGD.
• The key architectural characteristics of the convolutional layer is local connectivity and shared weights.

• Intuition: to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network, and hence to also control overfitting
• Pooling partitions the input image (or documents) into a set of non-overlapping rectangles (n-grams) and, for each such sub-region, outputs the maximum value of the features in that region.

For processing data with a grid-like or array topology:

• 1-D convolution: time-series data, sensor signal data

• 2-D convolution: image data

• 3-D convolution: video data

• The convolution, and pooling layers are typically used as a set. Multiple sets of the above layers can appear in a CNN design.
• After a few sets, the output is typically sent to one or two fully connected layers.
  • A fully connected layer is a ordinary neural network layer as in other neural networks.
  • Typical activation function is the sigmoid function.
  • Output is typically class (classification) or real number (regression).

• The final layer of a CNN is determined by the research task.
• Classification: Softmax Layer \[P(y=j|\boldsymbol{x}) = \frac{e^{w_j \cdot x}}{\sum_{k=1}^K{e^{w_k \cdot x}}}\]
  • The outputs are the probabilities of belonging to each class.
• Regression: Linear Layer \[f(\boldsymbol{x}) = \boldsymbol{w} \cdot \boldsymbol{x}\]
  • The output is a real number.

What hyperparameters do we have in a CNN model?

Main CNN idea for text:

Compute vectors for n-grams and group them afterwards

Example: “this takes too long” compute vectors for:

This takes, takes too, too long, this takes too, takes too long, this takes too long

https://arxiv.org/pdf/1408.5882.pdf

• MR: Movie reviews with one sentence per review. Classification involves detecting positive/negative reviews (Pang and Lee, 2005). url: https://www.cs.cornell.edu/people/pabo/movie-review-data/

• SST-1: Stanford Sentiment Treebank—an extension of MR but with train/dev/test splits provided and fine-grained labels (very positive, positive, neutral, negative, very negative), re-labeled by Socher et al. (2013). url: https://nlp.stanford.edu/sentiment/

• SST-2: Same as SST-1 but with neutral reviews removed and binary labels.

• Subj: Subjectivity dataset where the task is to classify a sentence as being subjective or objective (Pang and Lee, 2004).

• TREC: TREC question dataset—task involves classifying a question into 6 question types (whether the question is about person, location, numeric information, etc.) (Li and Roth, 2002). url: https://cogcomp.seas.upenn.edu/Data/QA/QC/

• CR: Customer reviews of various products (cameras, MP3s etc.). Task is to predict positive/negative reviews (Hu and Liu, 2004). url: https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html

• MPQA: Opinion polarity detection subtask of the MPQA dataset (Wiebe et al., 2005). url: https://mpqa.cs.pitt.edu/corpora/mpqa_corpus/

• A transformer adopts an encoder-decoder architecture.

• Transformers were developed to solve the problem of sequence transduction, or neural machine translation. That means any task that transforms an input sequence to an output sequence.

• More details on the architecture and implementation:

  • https://arxiv.org/abs/1810.04805

  • http://nlp.seas.harvard.edu/2018/04/03/attention.html

  • https://jalammar.github.io/illustrated-transformer/

There’s increasing evidence that pretrained models learn a wide variety of things about the statistical properties of language:

• Trivia: Utrecht University is located in …

There’s increasing evidence that pretrained models learn a wide variety of things about the statistical properties of language:

• Basic arithmetic: I was thinking about the sequence that goes 1, 1, 2, 3, 5, 8, 13, 21, …

There’s increasing evidence that pretrained models learn a wide variety of things about the statistical properties of language:

• Reasoning: Garry went into the kitchen to make some tea. Standing next to Garry, Carrie pondered her destiny. Carrie left the …

• Write with Transformer: https://transformer.huggingface.co/

• Talk to Transformer: https://app.inferkit.com/demo

• Transformer model for language understanding: https://www.tensorflow.org/text/tutorials/transformer

• Pretrained models: https://huggingface.co/transformers/pretrained_models.html

• Convolutional Neural Networks

• Transformers
  • “Small” models like BERT have become general tools in a wide range of settings
  • GPT-3 has 175 billion parameters

• These models are still not well-understood